Challenges to Replication and Iteration in Field Experiments: Evidence from two Direct Mail Shots

We conducted an experiment marketing microloans to farmers in the United States during spring 2015, and found that a simple direct mail letter increased borrowing from a government program. The subsequent spring, we built on this finding and enriched the design to test for information spillovers. The second experiment used a different sample frame and slightly different letter, due to operational considerations. The direct effect result did not replicate in the second year, thus lowering the likelihood that spillovers would be present and detectable. These results add to recent evidence on how (seemingly subtle) differences in context and treatment content affect consumer responses (e.g., Allcott 2015; Bertrand et al. 2010). At the same time, many treatment effects do seem to hold in multiple contexts (e.g., Banerjee et al. 2015). So what is an experimenter to do given the constraints on the number of factors that can be Challenges to Replication and Iteration in Field Experiments: Evidence from Two Direct Mail Shots